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CHAPTER  I 


INTRODUCTION 


I-l.  The  Mathematical  Programming  Problem 

This  report  is  concerned  with  properties  and  solution  methods  for 
ine(iuality-constrained  extremization  problems.  Generally,  the  problems 
can  be  put  as  follows: 

(l)  minimize  ^qCx)  constrained  by  f^(x)  ^0,  1  ^  1  ^  m, 
where  the  f^(0  ^  1  ^m)  are  real -valued  functions  defined  on  real 
n-space. 

Problem  (l)  is  referred  to  in  the  literature  as  a  mathematical 
program.  In  pstrtlculcu:,  (l)  is  a  linear  program  if  all  the  are 
linear  forms;  otherwise  it  is  a  nonlinear  program.  Hie  sinq[>lest  kind 
of  nonlinear  program  of  Interest  is  the  quadratic  program,  in  which  the 
minlmani,  f^  ,  is  a  convex  quadratic  function  and  the  other  f^  are 
linear  forms.  Vfhen  all  the  functions  in  (l)  are  convex,  it  is  called  a 
convex  program.  A  linear  program  may  be  regarded  as  a  special  type  of 
quadratic  program;  both  are  examples  of  convex  programs. 

Differentiability  of  the  f^  is  not  part  of  the  statement  of  (l), 
although  it  has  proved  to  be  a  useful  assumption  in  many  theoretical 
and  computational  investigations.  Among  studies  in  this  field,  the 
special  case  of  convex  programming  has  received  the  most  attention, 
probably  because  the  local  extrema  in  such  a  problem  are  always  global. 
In  all  the  problems  considered  here,  the  extremand  and  the  (.onstralnt 
functions  will  be  assumed  to  possess  at  least  continuous  first  partial 
derivatives;  however,  not  all  of  them  will  be  convex  programs. 
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1-2.  Results  Presented  in  This  Report 

The  mathematical  programming  problem  is  a  generalization  of  the 
classical  problem: 

(2)  minimize  constrained  by  f^(x)  =  0,  1  ^  i  ^  m 

which  is  usuailly  handled  by  the  method  of  Lagrange  multipliers.  This 
is  always  possible  when  the  mapping  f  ■  satisfies  a  regular¬ 

ity  condition  (7,  p. I98]. 

The  extension  of  the  Lagrange  multiplier  approach  to  the  matbematl- 
cal  programming  problem  was  first  accomplished  by  John  [28]  who  estab¬ 
lished  necessary  and  sufficient  conditions  for  a  solution.  Later,  Kuhn 
and  Tucker  [32]  presented  their  basic  work  giving  necessary  and  suffi¬ 
cient  conditions  for  a  nonnegative  saddle-point  of  a  differentiable 
function  and  their  relation  to  the  mathematical  progrananing  problem. 

They  obtained  necessary  conditions  for  a  solution  to  (l)  by  introducing 
a  regularity  assumption  which  made  it  possible  to  apply  the  Minkowski - 
Farkas  Lemma  [18,  p.  5]  and  thereby  produce  multipliers  like  John's  but 
with  a  leading  multiplier  of  unity.  John's  Theorem  emd  the  Kuhn-Tucker 
Theorem  on  necessairy  conditions  for  a  solution  to  (l)  are  discussed  in 
Chapter  II. 

The  well-known  duality  theory  of  linear  progrananing  was  formulated 
in  1947  by  von  Neumann  [54]  in  an  unpublished  manuscript  (see[8,  p.l25]) 
and  developed  by  Gale,  Kuhn,  and  Tucker  [25).  After  a  gap  of  several 
years,  the  notion  of  duality  was  extended  to  queuiratlc  progrananing  [12], 
[15],  and  to  convex  programming  [42],  [27].  For  quadratic  progrananing, 
the  proof  of  the  duality  theorem  can  be  obtained  by  applying  the  duality 
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theorem  of  linear  programming,  whereas  duality  In  convex  progrtmnlng 
has  usually  been  handled  by  an  application  of  the  Kuhn-Tucker  Theorem. 

The  dual  of  a  quadratic  or  convex  program,  as  originally  presented, 
differs  from  It  so  conspicuously  that  the  pair  of  problems  appears  to 
lack  the  beautiful  symmetry  of  the  analogous  problems  In  von  Neumann's 
formulation  for  the  llnesu:  case.  In  Chapter  III,  a  symmetric  formulation 
of  duality  for  nonlinear  programming  Is  presented  which  Includes  those 
above  as  special  cases.  Fart  of  this  chapter  Is  based  cn  the  author's 
paper  [6]  on  "Symmetric  dusil  qusulratlc  programs.  "  llie  extension  of  this 
Idea  to  the  nonlinear  case  Is  from  a  Joint  work  [11]  with  G.  B.  Dantzlg 
and  E.  Elsenberg. 

The  study  of  a  dual  peU.r  of  problems  often  leads  to  the  considera¬ 
tion  of  a  single  system,  the  coo^slte  of  the  two.  In  Chapter  IV,  the 
congposlte  problem  Is  viewed  as  a  special  case  of  the  general  program 
(5)  minimize  zW(z)  constrained  by  W(z)  ^0,  z  ^  0 

where  W  Is  a  mapping  of  N-spstce  Into  Itself.  Necessary  and  sufficient 
conditions  will  be  given  for  a  solution  to  (5)  when  W  Is  a  differentiable 
mapping  with  a  suitably  restricted  Jacobian  matrix.  Sevezal  important 
properties  of  the  Jacobian  matrix  of  the  mapping  W  in  (3)  are  left  In- 
varlsuit  by  an  operation  known  as  principal  pivoting  which  Is  an  exchange 
of  the  dependence  roles  of  certain  variables.  15ie  positivity  of  the 
determinant  of  a  (not-necessarlly  symmetric)  positive  definite  matrix 
turns  out  to  be  a  simple  consequence  of  the  results  on  principal  pivoting. 

Finally,  the  concept  of  a  positively  bounded  J8u:oblan  matrix  is 
presented  In  Chapter  V  in  response  to  the  need  for  a  sufficient  condi¬ 
tion  to  guau*antee  the  existence  of  a  solution  to  the  problem  (5).  The 
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existence  theorem  provides  at  the  same  time  am  algorithm  in  those  cases 
where  the  functions  and  values  it  demands  can  be  computed  and  actualiy 
inspires  the  algorithm  of  G.  B.  Dantzig  and  the  author  [9]  for  solving 
(5)  when  W(z)  =  Mz  +  q  and  M  is  a  positive  semi -definite  matrix.  A  new 
minimax  theorem  follows  from  the  main  existence  theorem. 

I-5.  Notations  and  Terminology 

There  is  a  fairly  standard  vocabuleiry  in  mathematical  programming 
which  facilitates  discussion;  we  shall  use  it  freely  here.  Given  the 
problem  (l),  the  mlnimand,  f^  ,  is  called  the  objective  function.  A  vec¬ 
tor  X  la  feasible  for  (l)  if  it  satisfies  the  side  conditions  (or  con¬ 
straints)  f^(x)  ^0,  ’  i  i  <.  The  constraint  set  of  the  problem  Is 
the  set  of  Its  feasible  vectors.  This  may  be  empty,  in  which  case  (l) 

Is  infeasible.  A  feasible  vector  Is  optimal  if  It  solves  the  problem. 

All  numerical  quantities  considered  here  belong  to  the  reals,  de¬ 
noted  R  .  Vectors  will  belong  to  finite -dlmenslonsG.  real  vector  spaces, 
n 

R  ,  and  whether  they  are  to  be  regarded  as  rows  or  columns  will  always 
be  clear  from  the  context  in  which  they  appear.  Thus,  for  example,  the 
expressions 


(4) 

X  =  (x^,... 

(5) 

Ax  =  b 

n 

(6) 

xy  -  ^  : 

i»l 

are  easily  understood.  In  (U),  x  Is  a  row  vector,  while  in  (5)  it  Is 
a  column  vector.  Equation  (6) --in  which  x  is  a  row  and  y  is  a  column- 
defines  the  symbol  xy  .  In  short,  no  special  notatlonal  provisions  will 
be  made  for  transposing  vectors.  We  may  not  treat  matrices  so  informally. 
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The  transpose  of  a  matrix  A  will  be  denoted  A'  , 

Vector  inequalities  will  be  used  extensively.  Let  x  ■ 
and  y  »  .  Then 

x^y  if  x^^y^,  l<.lin 
X  >  y  if  x^  >  y^  ,  1 

The  reverse  inequalities  ^  and  <  are  defined  analogously.  The  same  symbol, 

0  ,  will  be  used  to  denote  the  zero  vector  and  the  ordinary  scalar;  no  con¬ 
fusion  should  result  from  this.  A  vector  x  is  called  nc<nnegative  or  posi¬ 
tive  according  as  x^O  or  x>0.  A  nonnegative  vector  which  is  not 
0  is  called  semi -positive.  The  nonnegative  orthant  In  is  the  aet 
R^  consisting  of  all  its  nonnegative  vectors. 

If  f  is  a  differentiable  real -valued  function  on  an  open  subset  of 
^ ,  the  gradient  of  f  will  be  denoted 


_  dC  \ 

Vf  -  , 

and  if  it  is  twice  differentiable,  the  Hessian  of  f  will  be  denoted 

rfit  lil—  \ 

■  ■ 

In  dealing  with  differentiable  functions  of  two  vector  arguments,  it  will 
be  necessary  to  use  peurtial  gradients;  if  F(x,y)  is  such  a  function. 


^  F  ■ 


iL 


) 


„  V  n 


Also,  for  twice  differentiable  functions  F(x,y); 


p 


(n  X  n) 


A  square  matrix  M 


Vi,F 


V  F 
22 


of  order  r. 


dS- 

)  , 

(n  X  m) 

3^F 

)  , 

(m  X  n) 

A* 

)  , 

(m  x  m)  . 

will  be 

called  positive 

semi -definite 

If 

(7)  xMx  ^  0  all  X  c  . 

When  cquedlty  In  (7)  holds  only  If  x  ■  0,  M  is  positive  definite.  It 
Is  negative  semi -definite  (definite)  if,  and  only  If,  -M  Is  positive 
semi -definite  (definite). 


Although  It  Is  true  that  if 

M  -  (1/2)(M  +  M'), 

m 

then  M  is  synnetrlc  and 

xMx  ■  xMx  all  X  €  R*^  , 

it  will  not  be  possible  to  replace  M  by  M  because  M  will  represent 
a  nonsynmetrlc  Jacobian  matrix  needed  elsewhere. 

Finally,  a  word  about  the  reference  system.  The  manner  In  which 
chapters  and  their  sections  are  numbered  should  be  clear  at  this  point. 
Equations  are  numbered  consecutively  within  each  chapter.  Definitions, 
remarks,  theorems,  and  the  like  are  treated  as  equations.  A  reference 
to  an  equation  outside  a  given  chapter  will  be  made  by  prefixing  the 
chapter  number  to  the  equation  number.  This  rule  will  not  be  followed 
when  referring  to  ari  equation  in  the  same  chapter.  In  line  with  current 
practice,  numbers  in  square  brackets  refer  to  books  and  papers  listed 
at  the  end  of  the  report. 


CHAPTER  II 


NECESSARY  CONDITIONS  OF  OPTIMALITY 

II  -1 .  John ' 8  Theorem 

Ihe  earliest  resvlt  on  the  necessary  conditions  of  optimality  In 
the  problem  I-(l)  seems  to  be  that  of  John  [26,  Theorem  l].  He  was 
concerned  with  the  problem 

(1)  minimize  constrained  by  f(o,x)  ^0,  (a,x)  c  S  x  X 

where  S  is  a  coogpact  metric  space  and  X  is  a  subset  of  R^  ;  the 
partial  derivatives  of  f^^  and  f  with  respect  to  each  coiq>ODent  of 
X  are  assumed  to  be  continuous  on  X  and  S  x  X  ,  respectively. 

When  X  ■  R*^  and  S  Is  the  cocgpcct  metric  space  [l,...,m),  I'(l) 
can  be  viewed  as  a  special  case  of  II*  (l).  It  Is  convenient  to  give 
John's  proof  for  the  necessary  conditions  of  optimality  In  (l),  specializing 
it  to  the  mathematical  prograamlng  problem.  It  will  rest  partly  on  the 
following  statement,  which  is  due  to  Gordan  [2^]. 

(2)  Theorem.  L'^t  A  be  an  m  x  n  matrix.  Exactly  one  of  the  following 
alternatives  holds.  Either 

uA  ■  0 

has  a  seml*po8ltlve  solution  or 

Av  <  0 

has  a  solution. 

Gordon's  Theorem,  which  may  more  conveniently  be  found  in  [8,  p.136] 
and  [21,  p.i<6],  pre-dates,  by  about  twenty-five  years,  the  more  widely 
known  Minkowski -Farkas  Lensna  [I6,  p.  >]. 

The  proposition  we  want  is 
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(5)  Theorem  (John).  Let  x  solve  the  problem 

(4)  minimize  TqCx)  constrained  by  f^(x)  ^0,  1  ^  i  ^  m. 

A  A  A 

Then  there  exists  a  semi -positive  vector  . ,u^)  such  that 

(5)  u^f^(x)  *0,  1  ^  i  ^  m 
and  the  function 


(6) 


m 

4(x)  -  ^ 


(x) 


i-0 


has  a  critical  point  at  x  ,  1.  e. , 


(7) 


V  «(x) 
1 


J  u^TfJx)  .  0  . 


1=0 


Proof.  Define  the  sets 

(Q)  /'/I*  {l,...,m} 

and 

(9)  2-  (i  I  i  cM,  f^(x)  -  0)  . 

If  Z  is  empty,  let  »  1  and  u^  =  0  for  all  i  €  Indeed,  x  must 
belong  to  the  interior  of  the  constraint  set,  and  the  vanishing  of  the 
gradient,  VfQ  ,  at  the  unconstrained  local  minimum  implies  (7).  If  Z  is 
nonempty.  It  will  suffice  to  show  that  there  is  no  solution  v  to  the 
system 


(10) 


V  fQ(x)v  <  0 


V  f^(x)v  <  0  ,  i  e  Z. 


For  if  the  system  has  no  solution,  then  defining  u^  *  0  for  all  indices 
icM-Z  ,  there  exists,  by  Gordon's  Theorem,  a  semi -positive  vector 

A  A  A 

(uq,u^,  . . .  ,u^)  satisfying  (7),  This  vector  will  satisfy  (5)  ty  the 

definitions  made  above. _ 

1  The  sets  M  and  Z  will  be  used  again. 
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Now,  on  the  contrary,  suppose  (lO)  had  a  solution  v  .  Let  S(x,€) 

A 

denote  the  closed  ball  with  center  x  and  radius  €  >  0  .  There  exists 

A 

an  €  >  0  and  a  6  >  0  such  that  for  *tll  x  e  S(x,€) 


7fQ(x)v  +  6  0 

A 

7f^(x)v  t  6  i  0, 


1€  Z  . 


For  some  positive  T  , 


^^(x)  +  r  1 


Choose  a  >  0  satisfying 


& 


i€  M-  Z  • 


A  A 

Qinax(|  Vfj,  (x)v  |  |  i  e  fS/\  ,  x  €  S^x,e ))  <  T  . 

A 

The  compactness  of  S(x,€ )  and  the  continuity  of  the  partial  derivatives 
of  the  fj^  imply  the  existence  of  the  above  maximum.  For  some  0^  , 

0  <  0^  <  1  , 


Hence 


A  A 


f^Cx  +  Ov)  «  f^ix)  +  aVT^ix  +  0QOfv)v 


A  A 


f^(x  +  Ctv)  =  f^(x)  -f  aVf^(x  +  0^Ctv)v  ,  1  €  ^  . 


A  A 


f^Cx  +  Ov)  fQ(x)  -  06  <  f^{x) 

A  A  A 

f^(x  +  Ofv)  f^(x)  -  a6<  0  , 


A  A 


f^(x  +  Orv)  ^  -r  +  0(|  Vfj^(x  +  0^av)v  I  <  0  ,  1  €  M  -  Z  • 

A 

These  last  inequalities  contradict  the  optimality  of  x  in  (U),  and  so 

(10)  can  have  no  solution.  As  noted  earlier,  this  implies  (t). 

A 

(11)  Remark,  if  u_  is  positive,  it  may  be  assumed  to  equal  1.  If  the  vectors 


-9- 


Vf^(x),  i  e  ^  ,  are  positively  independent  (21,  p.  62],  l.e.  ,  linearly 
independent  over  the  nonnegative  orthant,  it  is  clear  from  the  theorem 
that  Uq  cannot  equal  0  . 

It  will  be  useful  to  record  the  conclusion  of  the  theorem  when  x 
is  required  to  be  nonnegative. 

(12)  Corollary.  Let  x  be  an  optimal  solution  of  the  program 

(13)  minimize  fQ(x)  constrained  by  x  >  0,  f^(x)  ^0,  1  ^  i  ^  ra. 

/V  A  A 

Then  there  exists  a  semi -positive  vector  (u.>u  ,...,u  )  such  that 

0  1m 

A  A 

(lU)  1  i 


u^Vf^(x)  >  0 


(16) 


Ill 


1  C  J  n. 


n  m 

If  f:P  -♦  R  is  a  differentiable  mapping  with  component  functions 


f,  ,. .. ,f  ,  let  J„  be  its  m  x  n  Jacobian  matrix: 
i '  '  m  f - 

(lY)  =  (ST  )  • 

cJ 

The  following  consequence  of  John's  Theorem  guarantees  the  exist- 

A 

ence  of  multipliers  in  (l4)  -  (16)  with  u^  ■  1  . 

A 

(18)  Corollary.  Let  x  be  an  optimal  solution  of  (15).  Suppose  there 

A 

are  p  columns  of  J^(x)  ,  say  ,  such  that  the  system  of  in¬ 


equalities 

(19) 


(x))  >  0,  u  ^  0 


has  only  the  trlviel  solution  u  »  0  .  Then  there  exists  a  semi -positive 
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vector  (u  .....u  )  -  (lo),  and  such  that  u  =  1. 

0  -L  m 

Proof,  Apart  from  tuc  stipulation  that  u  =  I,  such  a  vector  exists 

w 

/\  r\ 

by  the  previous  corollary.  If  u^  =  0,  then  the  vector  >  •  •  • 

semi -positive.  This  and  (l^l  imply  that  (19)  has  a  nontrivial  solution, 

A 

controi'y  to  our  ass'umption  Hence  u^  >  0  ,  and  tnerefore  may  be  taken  to 
equal  1. 

17-2,  Tne  Kuhn-lticker  Theorem 


The  Kuhn -Tucker  Theorem,  like  Jelin’s  Theorem.,  is  concerned  with 
necessary  conditions  of  optimality  in  the  mathematical  programming 
problem.  It  includes,  however,  an  extra  hypothesis  which  has  the  effect 
of  assuring  the  existence  of  multipliers  with  u^  =  1  . 

The  existeri'^e  cf  the  multipliers  in  John's  Tneorem,  above,  followed, 
via  Ckjrdan's  Tneorem,  from  .j’.e  fact  tna’^^  a  system  cf  linear  inecuolities, 
(10),  had  no  solution.  Guppose  w?  })ad  teen  able  to  show  that  the  less 


i-estrictive  system 

(20)  >,).  <  c 

Vf  .'x;v  ^  0,  i  £  z 


Thus,  fl,u^,...  ,a^)  would  te  senii -piosltive  and  would  correspond  to  the 
\ector  of  multipliers  in  John's  Iheorera.  The  difficulty  lies  in  the 
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fact  that  it  is  not  always  possible  to  show  that  (20 )  has  no  solution. 

The  example  (52,  p.484]  illustrates  this.  Let  fQ(x)  ■  -x^,  fj^Cx)  = 

_  A 

(x^  -  1)^  Xg,  f2(x)  =  -x^,  and  f^(x)  ■  -x^  .  It  turns  out  that  x  ■ 
(1,0)  is  the  unique  optimal  solution  to  problem  (6),  but  v  ■  (l,0) 
satisfies  (20)  where  we  note  that  'Z.  ■  {1,3). 

Some  regularity  condition  must  be  Imposed  if  we  wish  to  assert  that 

(20)  has  no  solution  when  x  is  optimal. 

(21)  Definition.  Let  x  be  a  boundai'y  point  of  the  constraint  set 

»  (x  |xeR^  f(x)  C  0) 

where  f  -•  R®  .  In  this  case,  the  set  Z  -  (1 1 1  €  M  -  0) 

is  nonempty.  The  (Kuhn-Tucker )  constraint  qualification  is  satisfied 
at  X  if  for  every  vector  v  satisfying  the  system  of  homogeneous 
linear  inequalities 

(22)  Vf^(x)v^0,  ieZ 

there  exists  a  continuously  differentiable  arc  a: [0,1]  -♦  such  that 
a(0)  a  X,  and  a'(0)  *  Xv  for  some  >  >  0.  (See  [52,  p.  ^5).) 

It  is  easily  shown,  along  the  lines  of  [12,  p.  I56],  that  the  con¬ 
straint  qualification  is  always  satisfied  at  the  boundary  points  of 
when  f  is  composed  of  llneeir  forms.  Arrow  and  Hurwicz  [1,  p.  2]  make 
the  interesting  observation  that  the  constraint  qualification  (21 )  is  a 
property  of  f  rather  than  of  .  They  show  that  two  distinct  mappings 
can  induce  the  same  constraint  set  and  yet  differ  with  respect  to  satis¬ 
faction  of  the  constraint  qualification. 

A 

(25)  Theorem  (Kuhn  and  Tucker).  Let  x  be  an  optimal  solution  of  (6). 

A 

If  X  belongs  to  the  boundary  of  ,  assume  that  the  constraint  qua].- 

A 

ification  is  satisfied  at  x  ;  then  there  exists  a  nonnegative  vector 
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/s 

u  =  (u^ 


(24) 


A 

.u  )  such  that 
*  m' 


m 


VfQ(x)  + 


PiVf,(x) 

inl 


0 


and 

(25)  u^fi(x)  =0,  1  e  M  . 

A  A 

Proof.  If  X  is  an  interior  point  of  ,  we  may  let  u  ■  0  ,  as 

A 

in  the  proof  of  (5).  If  x  is  a  boundary  point  of  ,  the  optimality 

A 

of  X  and  the  constraint  qualification  imply  that  (20 )  has  no  solution. 
For  otherwise  we  may  take  a  (and  \)  as  in  (21 )  and  deduce  that 


^(^0(0(0)))  -  9rQ(a(o))a'(o)  -  7fQ(x)Xv  <  0  . 

Since  a  maps  the  unit  interval  into  C^,  this  inequality  implies  that 

A 

there  exist  points  x  of  such  that  ^‘q(x)  <  fQ(x)  which  contradicts 

A 

the  optimality  of  x  .  Uierefore  (20 )  has  no  solution,  the  Minkowski - 
Farkas  Lemma  may  be  applied,  and  the  required  vector  exists. 

The  equations  (24)  and  (25)  are  usually  called  the  Kuhn-Tucker 
conditions  for  (6).  When  (6)  is  a  convex  program,  (24)  and  (25)  for 

A 

some  u  >  0  are  sufficient  conditions  for  optimality,  regeurdless  of  amy 
constraint  qualification.  (See  [32,  p.485].) 

In  some  cases,  the  Kuhn-Tucker  constraint  qualification  may  be 
difficult  to  verify.  But  satisfaction  of  this  regularity  condition  is 
merely  a  means  to  am  end,  namely,  the  Kuhn-Tucker  conditions.  We  shall 

A 

say  that  a  differentiable  mapping  f  -*  is  Lagrange  regxlar  at  x 

A 

if  for  some  nonnegative  u  ,  (24)  and  (25)  ^ure  the  necessary  conditions 
of  optimality  in  the  program  (6).  This  term  has  been  used  by  Arrow, 
Hurwi?2,  and  Uzawa  [2,  p.l8l]  in  their  study  of  the  interrelationships 
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between  some  of  the  veurlous  constraint  qualifications  which  have 


been  proposed  as  substitutes  for  that  of  Kuhn  and  Tucker. 

The  property  assumed  in  the  corollairy  (l8)  made  the  mapping  Lagrange 

A 

regular  at  the  point  x  .  All  the  programs  we  consider  in  this  paper  have 
either  linear  constraints  or  else  constraints  whose  Jacobiiin  matrix  at 
the  optimal  soJ.utlon  has  the  property  (l9);  in  either  case,  Lagrange 
regularity  is  at  hand. 

We  note,  in  passing,  that  the  necessary  conditions  of  optimality 

A 

in  (U)  reduce  to  the  Kuhn-Tucker  conditions  when  the  multiplier  Uq  is 
positive.  For  a  related  study,  in  which  the  positivity  of  the  multiplier 
associated  with  the  objective  function  is  crucial,  see  [4,  p.227]. 

Finally,  let  us  point  out  why  the  Kuhn-Tucker  conditions  are  pre¬ 
ferable  to  those  of  John.  The  answer  lies  in  the  Equivalence  Hieorem 
[52,  p. 486].  When  (15)  is  a  convex  program  for  which  the  constraint 

A 

qualification  (21 )  is  satisfied,  x  is  an  optimal  solution  if,  and  only 

A  A  A 

if,  there  exists  a  nonnegative  vector  u  such  that  (x,  u)  provides  a 
nonnegative  saddle-point  for  the  Lagrangian  function 


®(x,u) 


fo(x) 


1-1 
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CHAPTER  III 


SYJflETRIC  DUAL  PROGRAMS 

III-l.  Duality  in  Linear  Programming 
In  order  to  put  our  subject  in  i>er8pective,  we  recall  briefly  the 
symmetric  duality  theory  of  linear  programming.  Hils  involves  a  pair 
of  problems  such  as^ 

(1)  minimize  cx  constrained  by  Ax  +  b^O,  x^O 
and 

(2)  maximize  -by  constrained  by  -A'y  +  c  ^  0,  y  ^  0 

where  A  Is  an  m  x  n  matrix,  b  is  an  m-vector,  and  c  is  an  n-vector; 
all  three  are  constants,  whereas  x  and  y  represent  vector  variables. 

The  symmetry  of  this  pair  stems  from  the  fact  that  negation  and 
transposition  are  Involutory  operations,  (l)  is  called  the  primal  problem 
and  (2),  the  dual  problem.  This  terminology,  due  to  von  Neumann,  is  some¬ 
what  arbitrary  but  is  trawiitlonal  aund  will  be  used  here. 

Letting  P  and  Q  denote  the  constraint  sets  of  (l)  and  (2),  respec¬ 
tively,  the  major  theorems  on  duality  in  linear  programming  are: 

())  Weak  Duality  Theorem,  sup  -by  ^  inf  cx 

D  P 

(U)  Duality  Theorem.  If  either  program  in  the  dual  pair  has  an  optimal 
solution,  then  so  does  the  other,  and  when  this  is  so, 

max  -by  ■  min  cx  . 

D  P 

1  The  reader  already  familiar  with  duality  in  llneair  programming  will 
detect  that  (l)  aind  (2)  are  an  equivalent,  but  unorthodox,  statement  of 
the  dual  pair.  The  motivation  for  this  will  become  clear  later. 
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(j)  Complementary  Slackness  Theorem,  x  e  |  and  y  €  '_)  are  optimal 
solutions  of  the  primal  and  dual  problems,  respectively,  if,  and  only  if, 
(6)  y^(Ax  +  b)^  =  0,  1  ^  i 

(i)  Xj(-A'y  +  c)^  =  0,  1  ^  J  ^  n  . 

(b)  Existence  Theorem.  If  p  and  Q  are  nonempty,  then  both  problems 
have  optimal  solutions. 

(9)  Unboundedness  Theorem.  If  exactly  one  of  P  and  (3  is  nonempty, 
either  sup  -by  =  +  «  or  inf  cx  =  -  oo  according  as  P  or  is  empty. 

III-2.  Symmetric  IXial  Nonlinear  Pi'ograms 

Since  19^9,  the  notion  of  duality  has  been  extended  to  quadratic 

programming  by  Dennis  [12]  and  Dorn  [15]  and  to  convex  programming  by 

Wolfe  [U2],  Huard  [2?],  Mangasarian  and  others.  In  each  case, 

the  dual  pair  of  programs  lacks  s^/mmetry. 

The  object  of  this  section  is  to  present  a  treatment  of  symmetric 

dual  programs  [11  ]  which  will  cover  (l)  and  (2)  and  for  which  analogues 

of  at  least  (y),  (4),  and  (y)  can  be  proved.  In  the  case  of  quadratic 

programming,  all  five  of  the  corresponding  theorems  are  true.  In  the 

general  nonlinear  case,  (9)  is  false. 

S’uppose  K:R^xE^  -*  R  is  a  continuously  differentiable  function  and 
2 

considev  the  programs  : 

(10)  minimize  F(x,y)  =  K<'x,y)  -  yV^K(x,y) 

constrained  by  ^  ^  ^ 

(x,y)  >  0 

2  In  (10)  only  x  (in  (ll)  only  y  )  need  be  nonnegative. 
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(11) 


maximize  G(x,y)  =  K(x,y)  -  xV^K{x,y) 
constrained  by  7^K(x,y)  ^  0 

(x,y)  ^  0 

designated  primal  and  dual,  respectively.  Neither  one  is  necessarily  a 
convex  program.  Under  some  additional  hypotheses,  they  are  a  dual  pair, 
and  for  suitable  choices  of  K,  the  dual  pairs  previously  mentioned  are 
special  cases. 

As  in  III-l,  we  denote  the  constraint  sets  of  these  problems  by  P 
and  D  . 

Let  X  c  and  Y  c  be  nonempty  convex  sets.  A  function 
K:XxY  -•  R  will  be  called  convex -concave  if  K(  ,y)  is  a  convex 

function  on  X  for  each  y  €  Y  and  K(x,  )  is  a  concave  function  on 
Y  for  each  x  €  X. 

(12)  Weak  Duality  Theorem.  Let  K:R^xI^  -►  R  be  a  differentiable  convex - 
concave  function.  Then 

(13)  sup  G(x,y)  inf  F(x,y) 

D  P 

Proof.  Let  0  denote  the  empty  set.  We  adopt  the  convention  that 

(14)  sup  G(x,y)  a  -00  if  0  =  0  ,  inf  F(x,y)  =  +»  if  P  ■  0. 

It  therefore  suffices  to  assume  that  both  P  and  0  nonempty.  Let 

(x,y)  €  P  and  (x,y)  e  0  Since  K  is  a  differentiable  convex-concave 
function,  we  have  [29,  p.  405] 

m  0m  0m  0m  0»0  tm 

(15)  K(x,y)  -  K(x,y)  ^  (x  -  x)Vj^K(x,y) 

(16)  K(x,y)  -  K(x,y)  ^  (y  -  y)V2K(x,y). 

Subtracting  (l6)  from  (15)  and  rearranging  terms,  we  get 

(17)  F(x,y)  -  G(x,y)  ^  xVj_K(x,y)  -  ^  0. 
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(18)  Remark.  !nie  obvious  significance  of  the  weak  duality  theorem  is 
that  when  K  is  differentiable  and  convex -con cave,  (x,y)  c  P  and 
(x,y)  €  D  would  be  optimal  solutions  if  F(x,y)  and  G(5,y)  were 
equal.  We  note  from  (I?)  that^  if  this  were  the  case,  the  following 
would  hold; 

y^(V2K(x,y))^  =  0,  1  ^  i  ^  m 
Xj(Vj^K(x,y))j  =  0,  1  ^  J  ^  n. 

(19)  IXiality  Theorem.  Let  KtR^xR^  -*  R  be  a  twice  continuously  differen- 

tiable  function.  If  (x,yj  is  an  optimal  solution  of  the  primal  problem, 
(10),  «ind  Is  negative  definite,  then  (x,y)  €  D  and  F(x,y)  = 

G(x,y).  If  K  is  convex- concave,  (x,y)  solves  the  dual  problem,  (ll). 

Proof.  The  constraints  of  the  primal  problem  are  lagrange  regular 
at  (x,y)  because  the  hypotheses  of  II-(l8)  are  satisfied.  There  exists 
an  m-vector  v  ^  0  such  that  W2K(x,y)  =  0  and 

7^F(x,y)  +  V^K(x,y)v  ^  0 

5(ViF(x,?)  +V^K(x,?)v]  0 

V2F(x,y)  +  V22K(x,y)v  ^  0 

y(V2F(x,y)  +V22K(x,y)v]  =  0. 

When  F  is  replaced  by  its  definition,  these  reduce  to 

(20)  7lK(x,y)  V;^K(x,9)(v  -  J)  ^  0 

(21)  ^j^K(x,y)  +^^K(x,y)(v  -  y)  «  0 

^  If  the  scalar  product  of  two  nonnegative  vectors  is  zero,  then  each 
of  the  summands  in  the  expression  is  zero. 
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(22) 


V22K(x,y)(v  -  y)  ^  0 
(25)  y722K(x,y)(v  -  y)  -  0 

It  foUcwB  from  v  ^  0,  (22),  and  (23)  that 
(v  -  y)^^K(x,y)(v  -  y)  ^  0. 

This  Inequality  and  the  negative  definiteness  of  imply 


/s 

V  =  y. 

From  (20)and  (21) 

,  we  now  get 

(24) 

yj^K(x,y) 

^0 

(25) 

,'«7^K(x,y) 

*  0. 

Since 

A  A  A  /A 

V  =  y  and 

0 

u 

(26) 

CVJ 

*  0, 

Consequently,  (x,y)  €  0  and  F(x,y)  *  G(x,y)  «  K(x,y).  When  K  is 
convex-concave,  (x,y)  solves  the  dual  problem  by  the  weak  duality  theorem. 

(27) .  Remark.  A  similar  result  obtains  vben  (x,y)  solves  the  dual  prob¬ 
lem  and  7jj^K(.Y,y)  is  positive  definite.  In  both  theorems,  the  comple¬ 
mentary  slackness  conditions  are  visible  in  equations  such  as  (25)  and 
(26). 

For  suitable  choices  of  K,  the  symmetric  programs  (10)  azui  (U) 
reflect  the  form  of  certain  knovn  dual  pairs  of  the  programning  problems. 

For  instance  if 

(28)  K(x,y)  =  cx  -  by  -  yAx 

the  symmetric  dual  linear  programs  (1  )  and  (2)  result  from  the  operations 
indicated  by  (10)  and  (11),  respectively.  For  this  definition  of  K,  the 
hypotheses  of  the  weak  duality  theorem  are  satisfied.  On  the  other  hand , 
the  definiteness  assumption  of  our  duality  theorem  (19 )  carinot  be  satisfied 
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ly  tuch  a  function.  However,  ths  contribution  of  the  linearity  of  K 
in  each  of  its  variableo  is  sufficient  to  allow  weaker  hypotheses  for 
the  duality  theorea. 

Likeviee,  the  primal  convex  program 


i29)  minimize  ^0^^^ 

constrained  by  fj^(x)  ^0,  1  ^  1  ^  m 

X  ^  0 


has  the  dual,  according  to  Wolfe  [42], 


ra 


n 


m 


(50)  ma.xlmize  f^(x)  ^  ^ 


1-1 


>1 

m 


1=1 


constraint i  by 


vfo(x)  +  ^  0 


1=1 


y  >  0. 


Using  i.lO)  and  i'll),  this  pair  of  programs  can  be  obtained  from  the 
didferartiat  le  cor  ’’tx- con  cave  function 


m 

(51)  K(x,y)  -  f^(x)  +  (x). 

i=l 

Again,  the  derivation  of  tr.e  dual  pair  (29),  (50)  in  this  manner  is 
formal.  The  proof  that  if  x  is  an  optimal  solution  of  (29),  then 
t.here  existn  a  y  such  thuit  (x,y)  is  an  optimal  solution  of  (JO)  and 
such  that  the  extremal  values  of  the  objective  functions  are  equal  can 
be  obtained  from  the  Kuhn-Tucker  Theorem,  in  which  the  constraint  quali¬ 
fication  must  be  assumed.  Strong  assumptions,  like  those  of  (19),  are 
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made  by  Huard  [27]  and  Mangasarian  [33]  "the  converse  proof  of  the 

U 

duality  theorem. 


III-3*  Synmetric  IXial  Quadratic  Programs 
Let  A,  b,  and  c  have  the  meanings  assigned  In  III-l;  let  D 
and  E  be  symmetric,  positive  semi -definite  matrices  of  orders  n  and 
m,  respecti\  ely .  The  fianction  K  defined  by 

(32)  K(x,y)  =  ^-xDtx  +  cx  -  jyEy  -  by  -  yAx 

induces  s^Tnmetrio  dual  quadra^lc  programs  [6]  in  (1^)  and  (11).  When 
D  and  E  are  both  zero  matrices,  the  function  K  in  (32)  becomes  identi 
cal  to  that  defined  in  (28),  and  the  related  problems  reduce  to  symmetric 
dual  linear  programs.  When  D  or  E  is  a  zero  matrix,  one  gets  Dorn's 
dual  quadratic  programs  [13]-  See  Dorn's  paper  [11]  for  an  earlier 
effort  toward  symmetrizing  quadratic  programming. 

The  function  defined  in  (32)  isconvex-concave  [lo,  p.  5I]  and  has 
continuous  partial  derivatives  of  all  orders.  Therefore  the  weak  duality 
theorem  holds  for  this  K.  However,  7j^j^K(x,y)  =  D  and  '722K(x,y)  =  -E; 
these  matrices  are  positive  and  negative  semi -definite,  respectively,  and 
therefore  the  duality  theorem  (19)  does  not  apply  to  this  K.  Neverthe¬ 
less,  a  duality  theorem  can  be  established  and  several  other  interesting 
generalizations  of  the  duality  theory  of  linear  programming  can  be  made. 
This  will  be  done  in  the  pi'esent  section. 

1 

It  is  incorrectly  assumed  in  [33]  that  local  strict  convexity  for  a 
twice  continuously  differentiable  function  implies  the  positive  definite 
ness  of  its  Hessian  matrix  in  that  neignborhood.  This  is  false,  though 
its  converse  is  true  [29,  P-  ^^Ib]. 
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Explicitly,  the  dual  programs  are 


(35) 

minimize  F(x,y)  = 

^xDx  +  -^Ey  +  cx 

constrained  by 

Ax  +  Ey  +  b  ^  0 

(x,y)  ^  0 

and 

(5i^) 

maximize  G(x,y)  = 

-^ocDoc  -  -^yEy  -  by 

constrained  by 

Dx  -  A'y  +  c  ^  0 

(x,y)  2  0 

The  duality  theorem 

for  this  pair  runs 

as  follows: 

(35)  Duality  Theorem.  If  (x,y)  solves  the  primal  problem,  (55)»  then 
there  exists  a  vector  v  such  that  (x,v)  solves  the  dual  problem,  (3^), 
Ey  =  Ev,  and  F(x,y)  =  G(x,v).  Conversely,  if  (x,y)  solves  the  dual 
problem,  then  there  exists  a  vector  u  such  that  (u,y)  solves  the  primal 
problem.  Doc  -  Du,  and  F(u,  y)  =  G(x,y).  Moreover,  the  pairs  (x,v)  and 

/s  ^ 

(u,y)  solve  both  problems. 

Proof.  By  the  symmetry  of  the  problems,  the  converse  need  not  be 
proved.  The  Kuhn-Tucker  conditions  (after  some  simplification  and  elimina¬ 
tion  of  redundant  information)  read:  There  exists  an  m-vector  v  such 


that 

(36) 

Dx  -  A'v  +  c  ^  0 

(37) 

-  £^v  ^  0 

(38) 

x(lbc  -  A'v  +  c )  =  0 

(39) 

AS  As  As  . 

y(I^  -  Ev)  =  0 

(ho) 

v(Ax  +  +  b)  =  0 

(41) 

V  ^  0. 
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From  (56)  and  (Ul),  we  know  that  (x,v)  is  a  feasible  pair  for  the  dual 
problem.  To  show  F(x,y)  =  G(x,v),  it  suffices  to  show  F(x,y)  ^  G(x,v). 
By  (58)  and  (W),  this  amounts  to  showing 


(k2)  vEy  ^  fvEv  +  ?yEy. 

In  virtue  of  (59)>  this  is  equivalent  to  showing 
(43)  yEy  ^  vEv 

which,  however,  is  true  since  (39)>  (57),  (4l),  and  the  symmetry  of  E 
imply 

/^A  AA  AA  AA 

yEy  =  yEv  =  vEy  ^  vEv. 


Since  E  is  a  symmetric,  positive  semi-definite  matrix,  the  reverse  of 

A  A  A  A 

(42)  is  always  true,  hence  (y  -  v)E(y  -  v)  =  0,  and  this,  in  turn,  implies 

A  A  A  A 

[19,  p.  108]  Ey  =  Ev.  This  last  fact  shows  that  (x,v)  does  Indeed  solve 


both  problems. 

A  A  A  A 

We  note  that  with  (x,v)  as  above,  Ey  =  Ev,  and  the  complementeuy 
slackness  conditions  appear  in  (38)  and  (40), 

(44)  Remark.  This  duality  theorem  was  first  proved  by  showing  that  if 
(x,y)  solves  the  primal  problem,  then  there  is  a  certain  linear  program 
of  which  it  is  also  an  optimal  solution.  The  duality  theorem  of  linear 

A 

programming  was  then  invoked  and  the  required  vector  v  was  obtained. 
See  [6]  and  [13]- 

The  duality  theorem  (35)  shews  that  if  either  problem,  (33)  ox*  (34), 
can  be  solved,  then  P  and  D  (their  respective  constraint  sets)  are 

nonempty,  and  moreover  P  O  D  is  nonempty.  The  converse  of  this  is 


of  considerable  practical  significajice.  The  proof,  given  below,  will  de¬ 
pend  on  a  result  due  to  Frank  and  Wolfe  [19,  p.  IO8],  but  modified  here 


for  the  situation  at  hand. 

(45)  Theorem.  Let  f(x)  =  px  +  xCx  be  bounded  below  on  a  polyhedral 


convex  set  K  in  R  .  Then  f  assumes  its  infiraum  on 


ft  . 


Proof.  (Since  polyhedral  convex  sets  are  closed,  the  result  is  of 
particular  interest  when  R  is  unbounded.  )  There  is  no  loss  of  generality 
in  assuming  that  C  is  a  symmetric  matrix.  The  demonstration  is  by  in¬ 
duction  on  the  dimension  of  R  ;  there  is  nothing  to  prove  if  R  is  of 
dimension  zero.  Suppose  that  the  dimension  of  R  is  k  +  1.  It  is  pos¬ 


sible  to  write 


R  =  (s  +  rt 


seS^teT  >reRj 


where  S  is  a  bounded  polyhedral  convex  set  and  7”  is  the  intersection 
of  a  certain  polyhedral  convex  cone  with  the  unit  sphere  [24,  p.  44]. 
Note  that  for  all  triples  (r,t,'jr)  e  R  X  T  xR^,  r  +  Tt  €  R  and 

(46)  f(r  +  ft)  =  f(r)  +  f(p  +  2rC)t  +  f^tCt. 

Since  f  is  bounded  below  on  R  ,  it  follows  from  (46)  that  tCt  ^  0 
for  all  t  €  X  . 

If  tCt  >  0  for  all  t  y  ^  then  tCt  >  a  >  0  for  some  a  (T 


a  (T 


is  compact);  and  (p  +  2sC)t  >  S  for  some  S  which  may  be  assumed  to 
be  negative.  For  any  r=s+TteR  ,we  have 

(47)  f(r)  =  f(s)  +  r(p  +  2sC)t  +  f^tCt  >  f(s)  +  fS  +  r^a. 

The  minimum  of  the  quadratic  function  on  the  right-hand  side  of  (47)  is 
taken  on  when  f  =  -0/2a.  The  Infimum  of  f  over  R  is  therefore  at¬ 
tained  on  the  coraxjact  subset  (s  +  ft  |sgS  ,  t  €  T,  0  ^  r  ^  -6/2a} . 

Suppose  tCt  =  0  for  some  t  e  T .  Since  f  is  bounded  below  on  R  , 
(p  +  2rC)t  ^  0  for  all  t  €  T  •  li*  i'or  all  reR,r+ft€R  for 
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all  real  ir  (positive  or  negative),  then  (p  +  2rC)t  =  0.  In  this 
case  f(r  +  yt)  =  f(r)  for  all  r  €  R  ,  and  the  range  of  f  is 
unaffected  by  projecting  R  into  the  k-dimensicnal  subspace  orthogonal 
to  t  .  The  inductive  hypothesis  will  then  apply  and  yield  the  result. 

If  there  exists  (r,ir)  €  R  xR  such  that  r  +  ft  ^  ^  , 

consider  the  sets 

=  (r  I  r  e  R  ,  (p  +  2rC)t  =  0) 

Rg^IrjrcR  ,  (p  +  2rC)t  >  0)  . 

By  an  earlier  comment,  P  -  P  1  U  p  2  .  ^  the  argument  Just  given, 

the  infimum  of  f  on  R  is  attained  there.  For  re  R  2  » 

=  inf(r  I  r  +  ft  e  R  )  . 

For  each  r  e  R  2  »  0  ^  TTj.  >  -  w  .  Thus, 

b  =  r  +  r  t 

r 

belongs  to  the  boundary  of  R  .  Now,  for  each  re  R  2^ 

f(b^)  =  f(r)  +  r^(p  +  2rC)t  ^  f(r)  , 

which  means  that  the  minimum  of  f  on  R  ^  sought  on  the  bound¬ 

ary  of  R  .  Sat  on  each  of  the  k-dimensional  bounding  hyperplanes  of 
R  ,  f  attains  its  infimum.  This  completes  the  proof. 

(48)  Existence  Theorem.  If  both  R  and  D  nonempty,  then  the 
primal  and  dual  queuiratic  programs  {55)  and  (54)  have  optimal  solutions. 

Proof.  By  the  weak  duality  theorem,  the  primal  objective  function 
(which  is  quadratic)  is  bounded  below  over  P  .  Since  R  is  a  polyhedral 
convex  set,  the  theorem  (45)  applies.  The  remainder  of  the  assertion  is 
a  consequence  of  the  duality  theorem. 

To  complete  this  list  of  extensions  of  the  five  theorems  in  III-l, 
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we  mention  the 


(U9)  Unboundedness  Theorem.  IT  exactly  one  of  P  and  0  is  nonempty, 
either  sup  G(x,y)  =  ■♦•  »  or  inf  F(x,y)  =  -  oo  ,  according  as  P  or  0 
is  empty. 

Proof.  This  is  a  consequence  of  a  more  generaLL  result  on  programs 
with  linear  constraints.  See  [42,  Theorem  5]. 

III-4.  An  Equivalence  Theorem 

Once  more,  let  K:r'^xI^  -►  R  be  an  arbitrary  differentiable  convex- 
’  +  + 

>  n  m 

concave  function.  A  point  (x,y)€  called  a  nonnegative  saddle- 

point  of  K  if 

/S  /N  /V 

K(x,y)  ^  K(x,y)  ^  K(x,y) 

for  all  (x,y)  €  •  Kuhn  and  Tucker  [52]  gave  necesseo'y  and  sufficient 

conditions  for  a  nonegative  saddle-point  of  such  a  function: 


^  Ak  ^  ys  ^ 


(50) 

Vj_K(x,y)  >  0  , 

xV^K(x,y)  =  0  , 

X 

IV 

0 

(51) 

V2K(x,y)  0  , 

A  A  A 

y\;^Kix,y)  =  0  , 

A 

y  >  0  . 

Letting  P  and  0  he  the  constraint  sets  of  the  programs  (lO)  and 
(11),  respectively,  we  get  the  following 

(52)  Equivalence  Theorem.  The  point  (x,y)  e  P  O  L/  satisfies  the 

y^  /N  ys 

equation  F(x,y)  =  G(x,y)  if, and  only  if,  it  is  a  nonegative  saddle-point 
of  K  . 

Proof.  For  the  necessity,  refer  to  the  remark  (18).  The  sufficien¬ 
cy  part  is  immediate. 
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CIlAPTi^R  IV 


COMPOSITE  PROGR/J-IG 

IV-1.  Optimality  Criteria 

In  each  of  the  duality  theorems  established  above,  there  existed  a 
point  comi.ion  to  the  intersection  of  the  primal  and  dual  constraint  sets 
which  gave  equal  values  to  their  objective  functions.  It  is  natural  to 
form  a  program  whose  optimal  solutions  (if  any)  are  such  points.  This 
iS  done  by  means  of  the  composite  program : 

(1)  minimize  H(x,y)  =  xV^K(x,y)  - 

constrained  by  Vj^K(x,y)  ^  0 

-^2^(x,y)  ^  0 
(x,y)  ^  0. 

It  consists  of  minimizing  the  difference  between  the  primal  and  dual 
objective  functions  over  the  intersection  of  the  primal  and  dual  con¬ 
straint  sets. 

V7hen  K  is  defined  by  III-(52),  the  composite  program  for  quadratic 
programming  is 

(2)  minimize  xDx  +  yEy  +  cx  +  by 
constrained  by  Dx  -  A'y  +  c  ^  0 

Ax  +  Ey  -f  b  ^  0 
(x,y)  ^  0. 

We  shall  need  (2)  for  future  reference. 

By  defining  z  =  (x,y)  and  W(z)  =  (  ^K(x,y ) , -^K(x,y) ),  it  is 
clear  tiiat  (l)  has  the  form 
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(5) 


minimize  zW(z) 
constrained  by  W(2)  ^  C 


z  ^  0. 

With  N  a  n  m,  W  can  be  regarded  as  a  mapping  of  into  R^.  When 

K  is  convex-concave  and  twice  continuously  differentiable,  the  mapping 
W  »  (Vj^K,-72K)  is  continuously  differentiable  and  its  Jacobian  matrix, 


/  7j^j^K(x,y)  V^K(x,yj\ 

\^2j^K(x,y)  J722K(x,yj  y 


can  be  shewn  to  be  positive  semi -definite  for  all  (x,y)  e  • 

In  this  section,  we  study  optimality  criteria  for  programs  of  the 
form  (5)  where  W:R^  -•  R^  is  eua  arbitrary  continuously  differentiable 


map. 


In  (3),  the  objective  function  zV{z)  is  obviously  nonnegative  over 
the  constraint  set,  provided  it  is  nonempty.  Hence  any  feasible  vector 
which  makes  the  objective  function  vanish  must  bs  optimal.  This  much  la 
true  regardless  of  any  assuinptlons  on  W.  The  necessity  of  the  vanishiiig 
of  the  objective  function  at  an  optimal  solution  is  the  substance  of  the 
next  result. 

{k)  Theorem.  Let  z  be  an  optimal  solution  of  (3)  where  W  is  a  con¬ 
tinuously  differentiable  map  with  a  positive  semi-definite  Jacobian  matrix 
at  z.  Suppose  furthermore  that  the  constraints  are  Lagrange  regiJiLar  at 
z .  Then 


(5) 


zWlz)  =  0. 


Proof.  For  convenience,  let  M  =  e-ssumptlon  of 
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Lagrange  regularity,  there  exists  a  vector  ti  ^  G  such  that 

W(^)  +  M'(^  -  tl)  ^0 
z[W(‘S)  +  M’(z  -  Q)]  =  0 
=  0. 

These  facts,  the  feasibility  of  1,  and  the  positive  semi-lefiniteness 
of  M'  allow  us  to  construct  the  following  stririg  of  inequalities: 

(6)  0  ^  -  t)  ^  -  ^i)K’(0  -  *5)  ^  0. 

But  (6)  implies  (5). 

Thus,  under  the  assumptions  of  (h),  the  equation  (5)  ia  the  criterion 
for  optimality  in  (j). 

Another  version  of  the  same  theorem  involves  the  assuii5.tlon  that  the 
Jacobian  matrix  positive  principal  minors .  l.e.,  that  the 

square  submatrices  of  along  its  main  diagonal  have  positive  deter¬ 

minants  . 

(7)  Theorem.  Let  “2  be  an  optimal  solution  of  (j)  where  W  Is  a  con¬ 

tinuously  differentiable  map  whose  Jacobian  matrix  has  positive  principal 
minors  at  Then  its  constraints  are  Lagrange  regular  at  and  the 

objective  function  vanishes  there. 

Proof.  To  see  that  the  constraints  are  Lagrange  regular  at  'i,  ve 
use  the  hypothesis  that  Jy(^)  positive  principal  minors.  In  general, 

if  M  is  a  squeure  matrix  with  positive  principal  minors,  the  system  of 
homogeneous  linear  inequalities 

uM  ^  0,  u  ^  0 

has  only  the  trivial  solution  [22,  Theorem  1].  The  Lagrange  regularity 


-29- 


now  lollows  by  a  straightforward  application  of  II-(l8)  in  which 
f  =  -W,  X  =  z,  and  p  =  N. 

For  the  second  assertion,  we  again  let  M  =  J^(z),  and  let 
denote  its  i-th  column.  The  Kuhn-Tucker  conditions  componentwise  are 

W^(2)  +  m^(z  -  u)  ^0 
ra^(z  -  u)]  =  0 

=  0  1  ^  1  N 

Hence  for  each  i, 

0  ^  2^W^(z)  =  -2^m^(z  -  u)  ^  (u^  -  2^)m^(2  -  u). 

That  is, 

(z^  -  ti^)m^(z  -  u)  ^  0  1  ^  ^  N- 

But  since  M  (and  therefore  M')  has  positive  principal  minors, 

2  -  u  =  0  by  [22,  Theorem  2].  The  result  now  follows. 

(8)  Remark.  It  is  known  (see  for  example  [57])  that  positive  definite 
matrices  have  positive  determinants  and  consequently  have  positive  prin 
cipal  minors.  (We  offer  an  alterr.ate  proof  of  this  as  a  by-product  of 
some  results  in  section  IV-5. )  Of  course,  the  converse  of  this  proposi 
tion  is  false;  symmetry  is  required  as  an  extra  hypothesis. 

IV-2.  Quadratic  Programs 

This  section  is  devoted  to  an  existence  theorem  for  problem  (5) 
when  it  is  essentially  a  quadratic  program.  The  composite  problem  for 
quadratic  programming  was  stated  as  (2).  If  we  set 
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M  = 


q  =  (c,b)  ,  ar.d  z  =  (x,y) 


problem  (2)  becomes 

(9)  minimize  z[V\z  +  q) 

constrained  by  Mz  +  q  ^  0 

z  ^  0. 

This  is  an  example  of  (j)  where  W(z)  =  Mz  +  q  and  ‘^y(“)  = 

We  study  the  general  problem  (9)  under  the  assumption  that  either 
M  is  positive  semi-definite  or  M  has  positive  principal  minors.  With 
linear  constraints,  Lagrange  regularity  is  present,  and  the  optimality 
criteria,  (1)  and  (7),  are  in  full  force:  z  solves  (9)  If,  and  only  if, 
1(M5  +  q)  =  0. 

(10)  Theorem.  If  M  has  positive  principal  minors,  problem  (9)  has  an 
optimal  solution. 

Proof.  By  [22,  Corollary  2),  there  exists  a  vector  satisfying  the 
inequalities  Mz  >  0,  z  >  0.  A  suitably  large  positive  scalar  multiple 
of  any  such  z  will  be  feasible  for  (9)*  Since  z(Mz  +  q)  ^  0  for  all 
z  in  the  nonempty  polyhedral  convex  constraint  set  of  (9),  It  has  an 
optimal  solution  by  III-(U5). 

(11)  Remf^rk.  If  (9)  has  a  nonempty  constraint  set,  it  has  an  optimal 
solution,  regardless  of  he/  M  is  qualified.  But  in  general,  the  optimal 
solution  need  not  make  the  objective  function  vanish. 

We  emphasize  that  when  (9)  represents  the  composition  (2)  of  dual 
quadratic  (or  in  the  extreme  case,  linear)  programs,  an  optimal  solution 
of  (9)  solves  both  the  primal  and  the  dual  problems  Conversely,  if  an 
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optimal  solution  to  either  the  primal  or  the  dual  problem  exists,  then 
a  solution  to  (9)  exists.  In  this  sense,  they  are  equivalent.  However, 
the  composite  problem  has  certain  features  which  lend  themselves  to  a 
constructive  solution  technique,  i.e.,  an  algorithm,  which  will  be 
presented  in  Chapter  V. 


rv-J.  Principal  Pivoting 

In  problems  such  as  (3),  a  vector  z  is  sought  which  satisfies  the 
conditions 

(12)  W(z)  ^  0 

(13)  ^  ^  0 

(lU)  X  0. 

We  now  let  w  denote  an  N-vector  and  form  the  system  of  equations 
(15)  -  W^(z)  X  0  1  i  N. 

The  problem  ceui  be  viewed  as  one  of  finding  a  pair  of  nonnegative, 

orthogonal  vectors,  0  and  z,  satisfying  the  system  (15). 

N 

Let  us  assume  that  W  is  defined  on  all  of  R  .  The  N  variables 

w  in  (15)  eu*e  dependent ,  and  the  other  N  variables  z,  are  independent. 

In  the  language  of  linear  progranming,  the  w^  are  basic  variables  and 

the  z  are  nonbaslc  variables .  (See  [8].)  This  terminology  will  be 
J 

used  here.  A  basic  solution  to  (15)  is  one  in  which  the  nonbasic  variables 
all  equal  zero. 

For  the  remainder  of  this  paper,  we  denote  the  set  of  integers 
(1,...,N)  by  ^  .  For  i,j  € /V  »  ve  shall  define  =  3W^/Szy  This 

notation  is  suggested  by  the  analogy  with  the  special  case  W(z)  =  Mz  +  q. 
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If  then 

In  the  next  chapter,  we  shall  wish  to  reverse  the  dependence  roles 

of  certain  pairs  of  variables  in  the  system  (15).  The  idea  will  be  to 

select  a  certain  equation,  say  k,  in  (15)  and  "solve  it"  for  one  of  its 

nonbasic  variables  z  as  a  function  of  w  and  the  remaining  N  -  1 

J  ^ 

nonbasic  variables.  When  this  is  done,  we  shall  substitute  for  z  in 

J 

the  remaining  N  -  1  equations  and  obtain  a  new  system.  The  aim  of  this 
section  is  to  make  these  notions  more  precise  and  to  describe  the  relation 
between  the  Jacobian  matrix  of  the  original  mapping  and  that  of  the 
derived  mapping. 

We  recall,  first,  an  obvious,  but  useful,  fact  about  functions  of 
one  real  variable.  If  f :R  -•  R  is  a  continuously  differentiable  function 
with  derivative  f  satl6fyir.g  f'(x)^6>0(f'(x)^-6<0)  for  some 
6  and  all  x  e  R,  then  f  Is  a  strictly  increasing  (decreasing)  map  of 
R  onto  R. 

Now  suppose  that  for  fixed  k,J  tN,  there  exists  ft  >  C  such  that 

M 

W.  ,(z)  ^  ft  or  ,(z)  ^  -ft  for  all  z  e  R\  Then  the  k-th  component 

kj  Kj 


function  W^^  maps  R  onto  R.  In  particular,  for  each  Wj^  €  R  and 


each  (N-l) -tuple  (z.,...,z,  .,z.  €  R"  there  exists  a  u 

1  J-i  J+i  K 


nique 


z  6  R  such  that 
J 


Hence  there  exists  a  well-defined  function  W,  on  R  such  that 

k 


(16) 


if,  and  only  If, 
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Since  we  .  to  denote  basi-  and  nonbasic  variables  generically  by 


w  and  z,  respectively,  we  alter  the  notation  of  the  entire  set  of  2N 
variables  as  follows: 


Wj^  =  Zy  Zj  =  =  w^  (1  t/V  -  (It)),  z^  =  z^  (i  e  A/  -  (J))- 

With  the  new  notation,  (l6)  becomes 

\  =  0. 

After  we  define 


(18) 


Wi(s)  =  1  €  A/  -  (k), 


a  new,  equivalent,  system  emerges: 


(19)  =0,  i  e  Ay. 

The  system  (19)  is  said  to  be  obtained  from  (15)  by  a  simple  pivot 
{k,j).  The  mapping  W  is  continuously  differentiable. 

(20)  Definition.  A  principal  pivot  in  (15)  consists  of  a  finite  se¬ 

quence  ((k,j))  of  simple  pivots  such  that  both  projections  of  ((k,j)) 
into  equal  ,  for  some  S  Q  A/  .  If  5  =  (k),  we  shall  speak  of  (k,ic) 

as  a  principal  pivot  on  k.  We  shall  write 

(21)  W  =  P(W;((k,J>)) 

if  the  system  0  -  W(z)  =  0  is  induced  by  the  principal  pivot  ((k,j)} 
in  the  system  w  -  W(z)  =  0.  For  a  principal  pivot  on  k  we  use  the 
simpler  notation  W  =  P(W;k). 

If  W  =  P(Wj(k,j)),  the  Jacobian  matrix  J~  is  given  by  the  formulas 
(see  [7,  p.  118]) 
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(22) 


“kj  = 

“ij  ' 

®ki  '  “kl^^kj 

^ii  =  “u  -  f''iAAj>- 


i,i  €  N  -  (J,k) 


Now  let  (J  «=  P(W;k).  We  shall  need  a  relationship  between  the  princi¬ 
pal  minors  of  J-  and  those  of  J^,  for  which  we  Introduce  another  notation. 
If  M  is  an  arbitrary  matrix  of  order  N  andZcA/,  we  denote  by  (M)^  the 
principal  submatrix  of  M  formed  by  deleting  row  and  column  i  from  M  for 
each  1  €  /  .  The  empty  matrix,  (M)^  ,  will  be  admitted,  and  its  determinant 
will  be  defined  to  be  1.  Of  course,  M  =  (M)0* 

(23)  Theorem.  If  W  »  P(W;k),  then 

I  I  “  I  A{k)  l/'^kk 


for  all  X  £.  A/ 


where  vertical  bars  denote  determinant  euid  the  connective  A  denotes  the 

symmetric  difference.^ 

Proof.  The  statement  is  obvious  when  N  »  1.  Assume  N  >  1  and  (24) 
holds  for  all  smaller  positive  Integers.  It  will  suffice  to  prove  (24)  in 


(J5V 


and  I 

=  (k).  Now 
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^or  a  more  comprehensive  treatment  of  this  and  related  problems,  see 
Tucker's  paper  [40]. 


This  is  proof  of  the  first  part.  For  the  second  we  have 


0 

Wii 

• 

• 

l,k-l 

« 

« 

0 

• 

y 

l,k+l 

••• 

• 

®k-l,l 

• 

W 

k-l,k-l 

0 

k-l,k+l 

tJ 

k-l,N 

W 

kl 

kk 

y 

k,k-l 

kk 

1 

y 

k,k+l 

y 

kk 

y 

...  kU 

kk 

w 

k+1,1  *** 

• 

* 

k+l,k-l 

0 

• 

0 

k+l,k+l 

* 

• 

•••  ®k+l,t) 
• 

^N1 

w 

'^N,k-1 

• 

0 

• 

U 

N,k+1 

• 

"  NN 

« 

• 

w 

l.k-1 

• 

w 

Ik 

• 

• 

y 

l,k+l 

• 

'^IN 

• 

W 

k-l,k-l 

• 

w 

k-l,k 

• 

y 

k-l,k+l 

= 

W 

kl 

W 

kk 

k,k-l 

W 

kk 

1 

y 

k.k+l 

kk 

y 

... 

y 

'^kk 

W.  -  ,  •  •  • 
k+1,1 

« 

• 

w 

k+l,k-l 

• 

w 

k+l,k 

\+l,k+l 

• 

• 

•  •  -  W.  , 
k+l,N 

0 

• 

'^Nl 

• 

w 

N,k-1 

W 

Nk 

y 

N,k+1 

=  1 

These  equations  are  based  on  the  well-known  effects  that  elementary  rw 
operations  have  on  determinants.  (See,  for  example,  [5,  pp.  501-502].) 

As  a  direct  consequence  of  {2k),  we  have 
(25)  Corollary.  If  (J  =  P(W;k)  and  has  positive  principal  minors, 

then  so  does  Jg. 
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This  is  one  of  the  several  invariants  under  (simple)  principal 
pivoting.  Another,  which  we  state  without  its  simple  proof,  will  appear 
in  a  forthcoming  paper  by  Tucker  and  Wolfe  [Ul]. 

(26)  Theorem.  Let  W(z)  =  Mz  +  q.  Suppose  a  principal  pivot  in  the 
system  w  -  (Mz  q)  =  0  yields  the  system  w  -  (Mz+  q)  =  0.  Then  M 
is  positive  definite  (semi -definite)  if,  and  only  if,  M  is  positive 
definite  (semi -definite). 

Using  this  invariance  theorem,  we  can  prove  an  assertion  made  above 

(27)  Theorem.  The  determinant  of  a  positive  definite  matrix  M  is  posl 
tive. 

Proof.  Every  principal  submatrix  of  a  positive  definite  matrix  is 
positive  definite,  and  its  main  diagonal  consists  of  positive  elements. 
We  use  induction  on  the  order  of  M.  If  M  is  of  order  1,  there  is 
nothing  to  prove.  If  M  is  of  order  N  >  1,  a  principal  pivot  on  the 
index  1  yields  a  matrix  M  which  is  also  positive  definite.  By  the 
Inductive  hypothesis,  |  (M)^j  |>  0,  and  by  (24),  |M  |»  m^  ^ 

which  completes  the  proof. 

The  class  of  matrices  with  positive  principal  minors  Includes  all 
positive  definite  matrices,  whether  or  not  they  are  syranetrlc. 
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CHAPTER  V 


NONLINEAR  PROGRAMS  WITH  POSITBELY  BOUNDED  JAC03IANS 


V-1.  The  Main  Theorem 
According  to  IV- (10),  the  program 

(l)  minimize  2(M2  +  q) 

constrained  by  Mz  +  q  ^  0 

z  ^  0 

has  an  optimal  solution  if  M  has  positive  princitjal  minors.  (This  has 
also  been  established  constructively,  that  is,  by  an  algorithm  [9]*)  It 
is  natural  to  ask  whether  the  same  kind  of  theorem  can  be  proved  about 
the  program 


(2)  minimize  zW(z) 

constrained  by  W(z)  ^  0 

z  ^  0 

N  N 

where  it  is  assumed  that  W:R  -*  R  is  a  continuously  differentiable 

mapping  and  has  a  Jacobian  matrix  vith  positive  principal  minors 

N 

for  all  z  €  R  .  It  cannot. 

In  fact,  such  hypotheses  do  not  even  assure  the  existence  of  feasible 
vectors.  This  can  be  seen  in  the  one -dimensional  problem  with  W(z)  =  -e 
If  we  postulate  feasibility,  there  is  still  no  hope.  Recall  that  our 
optimality  criterion  says  that  a  feasible  vector  z  in  (2)  is  optimal 
if,  and  only  if,  zW(z)  ■  0.  Now  let 


^2^  ^2  ^ 


-e 


+  z 


2 


Then  the  Jacobian  matrix  is 
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The  prograni  is 


/  e‘^1  1 

\  0  1 

and  it  has  positive  principal  minors  for  all  z  e 
clearly  feasible.  If  ^  solves  it,  then 

z^{-e  1  +  z^)  -t-  =  0. 

^  -2 
Therefore  «  0;  but  the  only  finite  value  of  z^  for  which  -z^e  1 

can  vanish  is  0.  The  vector  ^^1*^2'  *  feasible,  since 

it  violates  the  constraint  W^(z^,22)  ^  0. 

The  seaurch  for  a  condition  on  the  mapping  W  which  would  guarantee 

the  existence  of  a  solution  to  (2)  has  led  to  the  following  notion. 

N  N 

(5)  Definition.  A  continuously  differentiable  mapping  W:R  -*  R  has 
a  positively  bounded  Jacobian  matrix  if  there  exists  a  number 

such  that  0  <  6  <  1  aind  such  that  for  all  z  c  every  principal 

minor  of  Jy(z)  li«fl  between  6  and  6*^, 

In  (2),  if  the  mapping  W  has  a  positively  bounded  Jacobian  matrix, 
the  optimality  criterion  is  the  same  since  the  Jacobian  has  positive 
principal  minors.  Furthermore,  if  W(z)  »  Mz  +  q,  Its  Jacobian  matrix 
is  identically  M  and  Is  positively  bounded  if,  and  only  If,  M  has 
positive  principal  minors.  Thus,  the  class  of  mappings  with  this  rather 
strong  property  is  nonempty.  By  considering  what  the  condition  means 
for  real-valued  functions  of  one  real  variable,  we  note  that  multivariate 
mappings  can  be  constructed  which  also  have  this  property. 

We  can  now  state  an  invariance  theorem  which  will  later  play  an 
Important  role. 

N  N 

(U)  Theorem.  If  W:R  -•  R  has  a  positively  bounded  Jacobian  matrix 
and  W  =  P(W;k),  then  W  also  has  a  positively  bounded  Jacobian  matrix. 
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Proof.  Thore  exists  a  real  nunber  6  such  that  0  <  6  <  1  a.^d 
for  all  z  e  and  all  I  ^N, 

6  ^  I  (J^.(z))  j  I  ^ 

It  follows  from  IV-(24)  that 

{JjU)),  I  i  6-2 

for  all  z  e  andJcA/,  hence  W  has  a  positively  bounded  Jacobian 
matrix. 

The  desirability  of  a  positive  lower  bound  for  the  partial  deriva¬ 
tives  (k  €  Ay  )  was  suggested  by  the  examples  above.  This  property 

need  not  be  preserved  in  the  absence  of  an  upper  bound  property.  For 
example,  if 

2  +2 

W(2.z)*2  +2 

2'  2^  12* 

we  may  choose  any  6  between  0  and  1  as  a  lower  bound  for  the  princi¬ 
pal  minors  of  Jy(z).  (Their  actual  lower  bound  is  1.)  But  they  have 

2  +2 

nc  upper  bound.  If  W  =  P(W;l),  we  find  that  =  l/(e  ^  ^  +  l) 

which  is  positive  but  not  bounded  below  by  any  positive  real  number. 

Thus,  the  positive  lower  bound  property  is  not  invariant  under  principal 
pivoting  unless  there  is  also  an  upper  bound  property.  This  example 
helps  to  motivate  the  definition  (3). 

(5)  Definition.  A  solution  (w,z)  of  the  system 

(6)  w^  -  W^(z)  =  0  1  €  As/ 

is  nondegenerate  if  at  most  N  of  its  2N  compOiiCnts  are  zero.  Otherwis 
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(7)  Remark .  There  are  perturbation  techniques  in  mathematical  programming 
for  ensuring  nondegeneracy.  One  of  them  involves  the  replacement  of  the 
real  variables  by  lexicographically  ordered  vectors.^  (See  [10]  and 
[8,  Chapter  10].  )  ^  vill  henceforth  be  assumed  that  all  solutions  to 

(6)  are  nondegenerate.  Nondegeneracy  implies  (a)  that  in  no  basic  solu¬ 
tion  of  (6)  can  there  be  a  basic  variable  with  value  zero,  and  (b)  that 
for  each  value  of  z^,  V»^(0, . . .  ,0,z^,0, . . .  ,0)  =  0  for  at  most  one  i  €  hJ. 

Recall  that  to  solve  (2j,  we  need  a  nonnegative  solution  (v,z)  of 
the  system  (6)  such  that  zw  =  0.  The  nonnegativity  of  (w,z)  corresponds 
to  the  feasibility  of  z. 

(6)  Definition.  The  number  of  negative  coordinates  in  a  solution  of  (6) 
is  called  its  Infeasibllity  index. 

A  basic  solution  to  (6)  with  an  infeasibllity  index  of  zero  solves 
problem  (2).  It  is  such  a  solution  ve  shall  show  to  exist.  This  will 
be  accomplished  by  an  iterative  procediire  which  does  not  employ  nonnegative 
solutions  to  (6)  until  the  last  step.  (Therein  lies  part  of  its  novelty.  ) 
But  with  a  scheme  of  this  kind,  the  "improvement"  in  successive  iterations 
must  be  measured  by  something  other  that  the  change  in  the  value  of  the 
objective  function,  zW(z).  We  do  this  by  obtaining  a  (finite)  sequence 
of  solutions  to  (6)  such  that  the  corresponding  Infeasibllity  indices 
form  a  monotonlcally  decreasing  sequence.  The  main  result,  then,  is  the 
following  existence  theorem. 

(9)  Theorem.  Let  W.-R*  -•  R  be  a  continuously  differentiable  mapping 
with  a  positively  bounded  Jacobian  matrix.  Then  there  exists  a  vector 

^  In  (l),  e.g.,  q  is  replaced  by  (q, l),  each  component  of  w,z  by  a 
vector. 
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2^0  such  that  W(z)  ^0  and  2W(z)  =  0. 

Proof.  It  will  be  convenient  to  make  the  notation  reflect  the 
Iterative  nature  of  the  method.  Therefore  we  replace  the  original 
system  (6)  by 

-  W^(z°)  =0  i  e  N 

Every  system  we  encounter  in  this  proof  will  be  derived  from  its  pre¬ 
decessor  by  a  simple  principal  pivot.  In  general,  the  system  after  the 
v-th  iteration  will  be  denoted 

w^  -  w^(2'^)  =  0  i  e  N 

where 

W^'  =  P(w''‘^;k^)  V  =  1,2, - 

At  any  stage;  If  (W^(0),0)  ^  0,  we  have  the  desli  :d  result.  If  this 

(basic)  solution  is  not  nonnegative,  It  has  a  positive  infeasibility 

index.  It  suffices  to  show  that  it  can  be  lowered  by  at  least  one  through 

a  .finite  number  of  principal  pivots,  for  the  Infeasibillty  index  of  the 

Initial  solution  (W*^(0),0)  is  at  most  N. 

Assume  w'^(O)  has  a  negative  component,  say  r.  Since  is 

positively  bounded  (recall  the  invariance  theorem,  (^)),  there  exists  a 

positive  number  6  such  that  ^  Hence  W^(0, . . .  ,0,Z^,0, .  .  ,  ,0, 

V  V 

=  0  for  seme  positive  value  Z  of  the  variable  z  .  Consider  the  set 

r  r 

V  V  V 

of  values  of  z  such  that  0  <  z  <  Z  and 
r  r  r 

wy(0,...,0,z'',0,...,0)  >  0  if  wy(0)  >  0. 

i  r  i 

This  bounded  set  is  closed  and  hence  compact.  Let  its  maxim'om  element 


-42- 


V  V  V 

be  denoted  C  .  We  shall  call  r  the  critical  value  of  z  for  this 
r  r - r 

iteration.  By  t^3  nondegeneracy  assumption,  it  is  positive  and  uniquely 

determines  an  index  s  such  that  W^(0, . .  .  . . .  ,0)  =  0.  Let 

=  P(w'';6).  If  8  =  r,  that  is  )  has  a 

lower  infeasibility  index  than  (w'^(0),0).  If  s  r,  then  the  in- 

v+1  V 

feasibility  index  of  the  solution  (W  (0, . . . . . . ,0), 

(0, . . .  . . .  ,0) )  does  not  exceed  that  of  the  previous  solution. 


We  now  consider  the  compact  set  of  z^ 
Cr  ^  ^  satisfying 


Balias  z'^)  such  that 
r 


w^'‘‘^(0,...,0,2^'*’^,0,...,0)  ^  0  If  w^(0, .  . .  . . .  ,0)  >  0. 

V+1 

Let  C  be  its  msucimum  element.  Then,  by  our  assumption  of  nondegeneracy, 
'r 

V-fl  V 

it  follows  that  ^  repeat  these  steps. 

At  every  iteration,  there  is  a  basic  set  of  variables.  Since  the 
critical  values  increase  strictly,  the  repetition  of  a  basis  would  imply 

V 

a  contradiction  in  the  critical  value  of  for  that  iteration.  There- 

th 

fore,  after  finitely  many  principal  pivots,  the  r  basic  variable  reaches 
the  value  0;  at  this  Juncture,  a  principal  pivot  on  the  index  r  lowers 
the  Infeaslblllty  index,  and  we  say  that  a  major  cycle  has  ended.  As  we 
remarked  earlier,  at  most  N  major  cycles  are  required  to  obtain  a  nonnega¬ 


tive  basic  solution  with  infeaslblllty  index  zero. 


V-2.  An  Algorithm  for  Quadratic  Prograamlng 
The  existence  theorem  (9)  provides  an  effective  algorithm  for  solving 
problem  (2)  when  the  required  values  are  readily  computed.  This  is  par¬ 
ticularly  true  when  W  is  of  the  form  W(z)  =  Mz  +  q  eind  M  has  positive 
principal  minors. 

In  this  section,  we  present  an  algorithm  for  solving  the  quadratic 
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program  (l)  under  the  assumption  that  M  is  positive  semi-definite. 


The  algoritium,  adapted  fron  [9],  is  analogous  to  the  method  of  existence 
theorem  Just  proved.  It  will  Involve  principal  pivoting  and  share  the 
characteristic  that  never  does  a  feasible  solution  appear  unless  It  is 
optimal. 

We  know,  IV-(ll),  that  If  (l)  is  feasible,  it  has  an  optimal  solution. 
But  it  Is  clear  that  (1)  need  not  be  feasible  if  M  is  merely  positive 
s^ml -definite;  our  algorithm  must  be  able  to  handle  this  contingency. 

We  therefore  '  igin  our  discussion  with  a  useful  observation. 

(10)  Remark.  If  the  matrix 


(U) 


'  “u 


) 


is  positive  semi -definite,  then 


(12) 


“12  *  “21  = 


This  is  true  because 

(13)  ^  ^  ^ ' 

However,  (I3)  can  hold  only  if  (12)  holds. 

This  remark  helps  us  to  prove 

(14)  Lenrui.  Let  (v,z)  -  (w,z)  be  a  solution  of  the  system  Iw  - 
(Mz  +  q)  =  0  where  M  is  a  positive  semi-definite  matrix  of  order  N. 
If  for  same  .r 


z  <  0,  w  <  0,  m  =0,  and  ra,  >0 
^  r  '  rr  ir  ^ 

then  the  system  has  no  nonnegative  solution. 

Pi  oof.  Since  m  =  C  and  rn,  >0,  for  all 
rr  ir  ^  ’ 

from  the  remark  above  that  m  <0  for  all  i  g 

ri  ^ 


all 


N, 

Now 


it  follows 


rn  .  z ,  +  Q  <  0 
ri  1  ^ 


IJ 

“r  =  I 

i=l 

Li 

implies  that  <  0  since 

i=l 
li 

"r  -  I  ”rlh  '  \ 

i=l 

can  have  no  nonr.e(3;.i.tive  solution;  tnv“refore  the  entire  system  has  no 
nonnegative  solution 

The  reason  for  assuming  z  ^  0  will  become  clear  in  the  sequel. 

Looking  aheaa  to  the  algorithm,  we  perceive  that  if  the  conditions  of 

the  Lemma  .no  critical  value  of  the  nonbasic  variable  z  can 

r 

be  found.  As  matters  stand,  the  converse  of  this  proposition  is  not 
true.  What  may  happen  is  that  the  entries  of  the  r-th  column  of  M 
possess  si?ns  which  agree  with  those  of  the  corresponding  basic  variables. 
These  entries,  being  the  partial  derivatives  of  the  basic  variables 
with  respect  to  z^,  make  it  impossible  to  ascertain  a  critical  value 
for  z  .  Under  such  circumstances,  z  could  be  increased  without  limit. 
However,  some  solvable  programs  have  this  property.  As  a  very  simple 
example,  let 

(15)  M  =  and  q  =  (-1,-2). 

The  vector  z  =  (0,2)  is  optimal  in  the  progriUn  (l)  formed  with  M  and  q. 

To  circumvent  this  difficulty,  we  introduce  an  artificial  lower 
bound  for  negative  basic  variables.  Let  ^  be  any  number  such  that 


m  .z.  >  0. 
ri  i  ^ 


But  the  equation 
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(16) 


-00  <  P  <  min  I  i  €  N  ) . 

If  the  minimum  on  the  right-hand  side  is  nonnegative,  then  the  zero 
vector  is  optimal  and  there  is  nothing  more  to  be  done.  The  other  possi¬ 
bility  implies  that  p  is  negative.  Throughout  the  algorithm,  p  will 
be  used  as  a  Icwer  bound  for  all  basic  variables  whose  values  are  nega¬ 
tive.  The  lower  bound  for  all  nonnegative  basic  variables  is  still  zero. 

At  this  point,  we  must  revise  our  notion  of  nondegeneracy. 

(17)  Definition.  A  solution  (v,z)  of  the  system  Iw  -  (Mz  +  q)  =  0  is 
nondegenerate  if  at  most  N  of  its  2N  components  have  the  value  0  or 
p.  Otherwise  it  is  degenerate. 

This  new  definition  extends  the  earlier  one  (^).  We  shall  assume 
that  all  solutions  to  the  system  Iw  -  (Mz  +  q)  =  0  are  nondegenerate. 

This  will  imply  that  if  (w,z)  is  a  solution  and  z  €  (0,0)  ,  tnen  for 
all  i  €  N  f  ^  (0,0).  If  for  i  eN  -  [r),  €  (0,0),  then  for  all 

values  of  z^,  W^(Zj^, . . .  ,z^_^,z^,z^^^, . . .  ,Zj^)c(O,0)  for  at  most  one  i  e  N/ . 
The  notation  will  be  analogous  to  that  of  (9).  We  replace  the 

0  ,00  Ov 

original  system  by  Iw  -  (M  z  +  q  )  =  0.  In  general,  either  the  system 
Iw  -(M  z  +q  )*0  will  be  obtained  from  Iw  -  (Mz  +q)  =  0 
by  a  principal  pivot  operation,  or  else  it  will  be  the  identical  system. 

The  steps  of  the  algorithm  are  listed  below;  a  more  detailed  discus¬ 
sion  follows. 

Step  0)  Set  V  =  0,  define  (v^,z^)  =  (q^,0)  and  define  0  as  any 
number  satisfying  (I6) 

Step  1)  If  (^,z'*)  ^  0,  stop.  The  solution  is  optimal. 

Determine  r  such  that  w'*  <0  or  z^  =  0. 

r  r 
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Step  2) 


satisfying 


V  V 

step  3)  Le't  Cj,  largest  value  of  ^ 

the  follcTwing  conditions: 


(i) 

z^  <  0 
r  ^ 

if  z*^  =  3; 
r 

(ii) 

W  '  z. , . 
r  1' 

-V  V  -V 

..|Z  .^Z  ^Z  *.1* 

'  r-1'  r'  r+x' 

if 

-V 

w 

r 

A 

0 

• 

(iii) 

• 

-V  V  —V 

...Z  ..Z  fZ 

'  r-1'  r'  r+1' 

if 

-V 

''i 

V 

0 

(iv) 

T,V/-V 
( ^2^  J  • 

-V  V  -V 

...Z  ..Z  .Z  ^1* 

'  r-1'  r'  r+1' 

if 

-V 

<  0. 

Step  4)  If  =  -Hx),  stop.  No  feasible  solution  exists. 

Step  5)  If  C)!  =  *^>  all  1  e  /V  -  (r), 

and  let  ) .  Return  to  step  1 

with  V  replaced  by  v  +  1. 

Step  6)  Let  s  te  the  unique  index  determined  by  the  conditions 
(ii),  (ill),  and  (iv)  in  step  3. 


Step  7) 


Step  8) 


Step  9) 
Step  10) 


If  =  0,  perfcr-Ti  the  principal  pivot  ( (r, e) ,  (s^r)) ; 


.  “V+1  “V  —^*'•♦•1  ,v  “  v+1  7V 

put  w  *z,w  -■ 

^  r  s'  8 


C'  ^  1  €  -  {r,6) 

and  then  )  for  all  1  eN  -  (r,8).  Return 

to  step  5  with  V  replaced  by  v  +  1  and  r  replaced  by  e. 

V  "^+1 

If  m  >0,  perform  a  principal  pivot  on  s.  Let  z  = 
rr  B 

,,V,-V  --V  -V  -V  ,  -V+1  ,,V+l,-V+lv 

W  (z.,...,z  ,z  ,,...,z„)  and  v.  «  W.  (z  J. 

s'  1'  '  r-l'^r'  r+1'  '  N  1  i  ' 

If  s  =  r,  return  to  step  1  v.th  v  replaced  by  v  +  1. 


If  8  ^  r,  return  to  step  3  with  v  replaced  by  v  +  1. 


This  completes  the  statement  of  the  algorithm.  Now  for  a  few  ex¬ 
planatory  remarks.  We  point  out  first  that  the  procedure  is  designed  to 
rule  out  any  increase  in  the  infeasibility  index  from  one  iteration  to  the 
next.  Every  return  to  step  1  is  accompanied  by  a  decrease  in  the  infeasi¬ 
bility  index  of  at  least  one.  The  only  possible  values  of  the  nonbasic 
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variables  at  a  retuni  to  step  1  are  0  emd  3  .  Consequently, 

^0  (in  step  l)  implies  z'*  =  0  and  hence  the  orthogonality 
of  the  pair. 

In  step  2,  such  an  r  exists  if  step  1  does  not  cause  termination. 

Not  both  8uid  sure  negative.  This  Is  a  property  of  the  initial 
solution  and  it  is  preserved  throughout  the  process. 

Step  3  is  clear. 

The  assertion  in  step  4  follovs  from  (14)  and  explains  its  hypothesis, 
z  ^  0. 

The  condition  of  step  5  implies  that  «  3.  No  pivoting  is  required 
in  this  case.  We  simply  increase  the  value  of  the  r-th  nonbaslc  variable 
and  adjust  the  values  of  the  basic  variables  accordingly.  Then  we  increase 
V  to  V  1. 

Step  6  makes  sense  because  the  conditions  (li),  (ill),  emd  (iv) 
determine  some  s;  it  is  unique  by  nondegeneracy. 

If  m'^  =  0,  then  r  s  in  stej.  for  the  definition  of  s  and 

the  nondegeneracy  assumption  imply  that  3w''/dz'^  is  not  zero.  Under  these 
circumstances,  we  cannot  make  a  principal  pivot  on  s  (as  we  would  have 
done  in  (9))*  We  have  =  m'^  <0  (either  (iii)  or  (iv)  determines 

S  Z*  8Z* 

s).  Hence,  by  (10\  m^  >0.  The  principal  pivot  {(r,8),(s,r))  can  be 

s 


achieved. 

Recall 

that 

v+1 

w 

r 

V 

v+1 

w 

s 

V 

=  z  , 
r' 

v+1  V 

w  =  w 

i  i 

all 

N. 

lr,8} 

V-fl 

z 

r 

=  w'^ 
s' 

v+1 

z 

s 

V 

=  w  , 
r' 

v+1  V 

Zi  =  z^ 

all 

1  £  /V  - 

{r,s). 

The  values  they  acquire  in  the  new  solution  are  given  as  follows; 
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The  Instruction,  In  step  7,  to  return  to  step  vlth  v  replaced 
by  V  -f  1  emd  r  replaced  by  s,  suggests  an  atteapt  to  Increase 
(alias  v^)  fron  Its  negative  value.  It  vas  this  variable 

6  IT 

V+1 

which  ve  originally  wanted  to  Increase  toward  zero.  Notice  that  m  >0, 

08 

so  that  (alias  z^)  will  not  be  affected  by  the  forthcaad.ng  Increase. 

6 

The  principal  pivot  Indicated  in  step  8  Is  possible  because  m  >0. 

88 

If  the  alternative  corresponding  to  step  9  1b  the  case^  the  awanlng 

V 

of  the  previous  step  Is  that  the  negative  variable  w^  has  risen  to  0, 

as  we  wished.  In  other  words,  s  was  deteroined  by  (ll).  In  the  new 

solution,  we  shall  have  ■  0  and  w^'*’^  >  0. 

*  r  r 
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From  step  10,  we  re-enter  step  5  eind  increase  the  r-th  nonbaslc 
variable  further.  An  important  point  is  that  after  the  principal  pivot 
on  s  ,  we  get 


-m 

v+1  sr  ^ 

m  =  — —  >  0  . 

sr  V  ^ 


m 


ss 


v+1 


Hence  the  s-th  basic  variable  will  Increase  as  is  Increased. 

r 

We  eure  now  ready  to  argue  the  finiteness  of  the  process.  As  we 
have  already  stated,  every  return  to  step  1  entails  a  reduction  in  the 
infeasibility  index.  Since  it  never  increases  during  tbe  process,  only 
finitely  many  returns  to  stepl  can  occur.  (These  correspond  to  major 
cycles  in  (9)*)  0nJ.y  finitely  msuiy  returns  to  step  3  cem  occur  between 

returns  to  step  1.  If  step  2  Is  entered  at  iteration  v  ,  a  peurticular 

V  ^ 

0 

index  r  is  determined,  and  step  3  that  be  increased.  In 

^  V 

each  iteration  preceding  a  retvum  to  step  1,  the  variables  w  ^  and 

V  ^ 

0 

z  each  Increase  mooctonlcally,  and  their  sum  Increases  in  a  strict 

^  V  V 

'O 

sezise.  The  N  -  1  nonbaslc  variables  distinct  fron  z  (or  w  , 
whichever  happens  to  be  nonbaslc)  all  have  value  0  or  ^  .  These 
facts  preclude  the  possibility  of  Infinitely  many  Iterations  before  a 
return  to  step  1.  Consequently  the  process  is  finite. 


V-3.  An  Application  of  the  Main  Theorem 
The  existence  of  theorem  (9)  Is  applicable  to  the  noonegative  saddle - 
point  problem  (see  III-4). 

(l8)  Theorem.  Let  K:R*^xH®  R  be  a  twice  continuously  differentiable 
function  and  suppose  that  ("^K^-V^K)  has  a  positively  bounded  Jacobleua 
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matrix.  Then  K  has  a  nonnegative  saddle-point  (x,y). 

Proof.  We  use  the  inteiTJretation  of  (2)  as  the  composite  program 

A  A 

IV-(l).  By  (9),  there  exists  a  point  (x,y)  such  that; 

A  A  AAA  A 

(19)  7j^K(x,y)  ^  0,  =  0,  x  ^  0; 

A  A  ^  A  A  A 

(20)  72^(x,y)  ^  0,  y^^Kix,y)  =  0,  y  ^  0. 

Since  K  is  twice  continuously  differentiable,  .^K  ani  “^22^ 
symmetric;  they  both  have  positive  principal  minors.  Hence  both  are 
positive  definite.  This  implies  that  K  is  strictly  convex-concave 
[29,  p.U)6].  The  proof  is  complete  since  (19)  and  (20)  are  the  necessary 
and  sufficient  conditions  for  a  nonnegative  saddle-point  of  a  differentiable 
convex- concave  function. 

We  mention,  in  closing,  that  for  this  point 

A  A 

max  min  K(x,y)  «  min  max  K(x,y)  =  K(x,y) 
y^  x^  x^  y^ 

Therefore  (16)  may  be  construed  as  a  minlmax  theorem.  Many  such  theorems 
may  be  found  in  the  literature  [17],  (50],  [59]*  Although  many  Involve 
convex -concave  (but  otherwise  more  general)  functions  K,  compactness 
appears  to  be  an  essential  ingredient  in  the  hypotheses.  It  Is  felt 
that  the  noncompactness  of  and  Justifies  the  restrictive  hy¬ 
potheses  of  (l8). 
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